Thank you @rjzamora for suggesting this idea. I think a sparse host buffer data source could make sense in libcudf.

The sparse host buffer data source is meant to be an improvement over the current approach which prepares a host buffer data source for reading with the bytes partially populated and the rest uninitialized. The current approach carries a risk for data corruption if libcudf ever reads an uninitialized byte range, whether from a bug in read_parquet or an over-read in JSON/CSV byte range support. If we had a sparse host buffer data source, it could throw if data were accessed outside of valid byte range. There could also be a performance benefit and host memory pressure reduction in using a sparse host buffer data source because we could allocate smaller host buffers with less wasted space.

• Would you please help us measure the benefits of a sparse host buffer data source? For this test we could compare the read throughput of a 500 MB file versus a 5 GB file with only 500 MB populated.
• We might want to separate a "single range" sparse host buffer from a "multi range" host buffer.
  • A sparse host buffer representing a parquet file will have a footer byte range, and then byte ranges for each column chunk (+metadata). This sparse host buffer would have multiple ranges, and we would expect read_parquet to never read outside a populated byte range.
  • A sparse host buffer representing a CSV/JSON file (I believe) must be constrained to only use a single byte range, and the range should be sized to allow over-reading to find the end of the last record. If we had multiple byte range in a single sparse host buffer, I'm not sure how that would interact with byte-range reads over the sparse host buffer... (but it would probably be a nightmare). It's likely that sparse host buffer data sources for text files will have a single range anyways.